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ABSTRACT 

This investigation documents the impact of the Van 
Leer Study, a large-scale evaluation study of achievement in the 
primary schools of Israel. It is intended to increase understanding 
of the process of evaluation utilization, showing how evaluation 
findings and other kinds of information can work together, over time 
and in a variety of ways, to influence decision making. The Van Leer 
Study provided a comprehensive review of three major areas: resources 
and their allocation among the different types of Israeli public 
schools; student achievement in reading, arithmetic, Biblical 
studies, geography, and science; and home and school variables 
associated with achievement. The study was designed for multiple 
clienteles and was reported in a variety of formats. When released, 
the evaluation report generated considerable controversy among its 
major audiences. The ultimate impacts of the lengthy report were that 
the various groups were led to a closer examination of the issues; 
that a number of Parliamentary recommendations were made; that a 
variety of program changes were initiated at the Ministry of 
Education; that additional research in the form of replication 
studies and supplementary analyses resulted; and that further study 
is currently ongoing. (PN) 



******************************************************* 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
*********************************************************************** 



9 

ERIC 



\ 



X 



U.S. DEPARTMENT OF EDUCATION 

# NATIONAL INSTITUTF OF EDUCATION 

EDUCATIONAL HFSOURCES INFORMATION - 
CENTER {ERIC) 
^This document has been reproduced as 
received from the person or organization 
originating it 

Minor changes have been made to improve 
reproduction quality 

IMPACT OF A MAJOR NATIONAL EVALUATION STUDY: 

• Points of view or opinions stated in this docu- 

ISRAEL'S VAN LEER REPORT* mcnt d ° not "««wn»y represent oH.c.alNIE 

Lf\ 

PO Marvin C . Alkin -permission to reproduce this 

nX> Center for the Study of Evaluation, UCLA MATERlAL has been granted by 

O and ! M ,C . f\IA:~ 

Arieh Lewy ' 

q Tel Aviv University 

. . TO THE EDUCATIONAL RESOURCES 

INFORMATION CENTER (ERIC)." 

Introduction 

This investigation documents the impact of the Van Leer Study, 
a large-scale evaluation study of achievement in the primary 
schools of Israel. It is intended to increase understanding 
of evaluation utilization, showing how evaluation findings and 
other kinds of information can work together, over time and in a 
variety of ways, to influence decisionmaking. The data sources 
included all relevant primary-source documents reflecting the 
reactions of individuals, groups, agencies, the press, and the 
general public to the Report, as well as research publications 
that cited the Van Leer Study. 

General Features of the Van Leer Study 

iTlT^T^nTor faculty member of the Hebrew University 
School of Education applied for a grant from the Van Leer Foundation 
to conduct a large-scale survey of the Israeli primary schools. 
The grant was awarded in 1970, the Ministry of Education provided 
supplementary funds in 1972, and the final report was released in 
1977. There was widespread agreement that such a study was 
needed, since the annual SEKER examination (an omnibus achievement 
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test given in the last year of primary school) was abolished in 
1971, creating a demand for an alternative information source on 
achievement in the primary schools. 

The Van Leer Study was designed to provide a comprehensive 
picture of the Israeli primary school system in three areas: 
(1) resources (facilities, budget, manpower) and their allocation 
among various types of schools; (2) student achievement in five 
subjects (reading, arithmetic, biblical studies, geography, and 
science); and (3) home and school variables associated with 
achievement. Thus, though the study took the form of an assess- 
ment (a summary of achievement measures) , it had evaluative over- 
tones; that is, both the authors and the sponsors of the study ex- 
pected it to yield inferential judgments about the effectiveness 
of the Israeli educational system. Of special interest was the 
success of continuous efforts to raise the achievement level of 
children of Asian-African origin and to reduce the achievement gap 
between these children and children of European-American origin. 

Although the Report was prepared in a form which fit the 
needs of the research community rather than of lay readers, the 
authors had multiple clienteles in mind. To assure that a wide 
audience was reached, both the Principal Investigator and a 
Ministry of Education spokesperson prepared summaries. Other 
audiences were reached through oral presentations. Moreover, 
prior to publication, brief progress reports were issued from time 
to time, creating awareness of the importance of the study and 
arousing interest in its results. 
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The van Leer Study was initiated at the highest executive 
ievel of the educational hierarchy- the Director General of the 
Ministry of Education-and was carried out by an independent re- 
search team from a university. Thus, many top officials in the 
system felt that they were the target of the evaluation; that xs, 
that the results would constitute a kind of verdict upon their 
actions. 

The target population comprised pupils in the first, second, 
fourth, and sixth grades of the Jewish State School System. A 
stratified random sample was drawn from this population. The total 
sample consisted of 98 schools, 614 classes, and 17,700 pupils. 
Two types of variables were considered: (1) student variables 
(family background, personal characteristics, achievement) and 
(2) school variables (teacher and principal characteristics, school 
programs, physical conditions,. Data were collected by means of 
a variety of instruments, including questionnaires and both ability 
and achievement tests. 

The Van Leer Study Findings 

The Report itself comprises 18 chapters. Chapter 1-4 are 
introductory, describing the study's background, objectives, and 
methodology. Chapter 5-8 present general information on: famrly 
characteristics: school facilities, programs, and resource alloca- 
tions (and their correlation with achievement, , the backgrounds 
and attitudes of teachers and principals: and the personality charac- 
istics, motivation, and attitudes of pupils. The central 
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section (Chapters 9-14) deals with student performance on various 
ability and achievement tests. in addition, data are presented 
for pupil groups classified along two dimensions: (1) ethnic 
origin and generation, and (2) parents' educational level. In 
discussing test results, information was provided on the propor- 
tion of students making scores of at least 60 percent correct re- 
sponses (considered by the PI to constitute a "pass" grade). 
Chapters 15-17 use the achievement data as a basis for examining 
interrelations between achievement and other criterion variables: 
Chapter 15 compares level of achievement in various types of 
schools (e.g., religious vs. nonreligious) ; Chapter 16 presents 
a hierarchical model for partitioning the variation in reported 
achievement test scores; and Chapter 17 pulls together all data 
bearing on the impact of integration. Chapter 18 is summary, die- 

cussion, and conclusions. 

in the Principal Investigator's Summary of the Report, find- 
ings were clustered into three groups: 

o Equality of resource allocation : There was still a gap 
between schools for advantaged and disadvantaged learners 
with- respect to both student achievement and resource al- 
location, despite Ministry efforts to increase resource 
allocations to "disadvantaged" schools. One conclusion 
was that "advantaged" schools supplement their budget with 
parental donations and are more alert in taking advantage 
of available funds, whereas lack of adequate staff at 
"disadvantaged" schools prevents them from benefiting 
fully from the grants allocated to them. 



9 

ERIC 



5 



o Achieve^^ GradeS WGre aSSignSd 

only to individual pupils and to student groups but also 
to the system as a whole: in reading, good; in mathematics 
and biblical studies, satisfactory; and in science and 
geography, unsatisfactory, 
o Achievement gap between ethnic groups: Children of Asian- 
African origin still achieved at a much lower level than 
children of European-American origin; those of mixed 
parentage fell between the two groups. The achievement 
of third-generation Israelis (i.e., those whose parents 
were born in Israel) was higher than that of second-generation 
Israeli pupils (those whose parents had immigrated). 
The Chief Scientist of the Ministry of Education, in a written 
critique, praised the study generally but criticized several 
methodological procedures and questioned the validity of some of 
the findings. This criticism touched on two main points: (1) the 
conclusions drawn from achievement differences between generations 
of immigrants (the Chief Scientist said that alternative explana- 
tions had not been ruled cut: for instance, later immigrants may 
come from lower educational levels than earlier immigrants, : and . 
(2) the legitimacy of making comparisons across subject areas (i.e., 
stating that pupils had performed better in one subject than in 
another. 

The Principal Investigator defended the Report, the Chief 
Scientist replied, and the debate occasionally tocx on a personal 
tone; this aspect received excessive attention from the press. 
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The release ef the Report was eagerly awaited both by the 
research and educational communities and by the general public. 
Shortly before its release, two seminars (both on educating dis- 
advantaged students, were held, calling further attention to it, 
though neither seminar was directly addressed to the Report, some 
of the findings were leaked at these seminars, thus creating more 

• „„ ft,. Raoort still faced two hurdles: 
anticipation. Once in print, the Report 

(1) the parliamentary elections that took place in March 1977 , 
release was delayed so that the Report could not be exploited for 
campaign purposes, and ,2, the time needed for the Chief Scientist 
to review and critigue the voluminous Report. Some newspapers 
accused the Ministry of trying to pigeonhole the Report because 
of findings unfavorable to the educational system. Thus, when it 
finally was released, the Report became a best seller, and the 
first 1,000 copies disappeared rapidly. 

The Impact of the Va n Leer Study. 

Early reactions to the Report came from various groups, all 
of which tended to look at the study not for what they could learn 
from it but for what it told them about their own pet issues. 
They tended to distort the findings to support their own particular 
views. The study findings elicited frustration because they 
were taken to indicate lack of progress in the school system. 

Newspaper stories tended to be sensational rather than 
analytic. Two examples show the press' distorted treatment of 

, First the press labeled the religious schools 

the study findings. First, tne p« 

"inferior" to the nonreligious schools because of raw score 
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differences in student achievement, without taking into account 
differences in the student input of the two types of schools. 
Second, differences in achievement between integrated and non- 
integrated schools were used as evidence against the merits of 
integration. 

On June 27 and 29, 1977, about a month after the Report was 
released, the Knesset devoted a debate to the implications of the 
study. The political context was complex and delicate, since a 
coalition government had just taken office and a new Minister of 
Education, from the National Religious Party, had been installed. 
Thus, the debate took on political overtones, when two members of 
the left wing of the Labor Party expressed the fear that the new 
Minister would use the study findings to increase the time de- 
voted to religious study in the schools, taking time away from 
secular studies. The Knesset debate relied heavily on comparative 
data about various grade levels and subjects, even though the 
Chief Scientist had criticized these findings. The character of 
the debate illustrates that, when readers do not fully understand 
the presentation of findings, they will substitute their own 
simplified version. In other words, scientific jargon invites 
erroneous interpretations. 

The Pedagogical Council of the Teachers Union, which concerns 
itself with all matters pedagogical, devoted several sessions of 
its annual seminar to discussing the implications of the Report, 
inviting presentations by the research team and the Chief Sci- 
entist. Questions raised at these sessions centered on those 
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policy implications of particular interest to the participants: 
e.g., grade repetition, ability grouping, extension of the school 
year. The substantive findings tended to be ignored. 

After the first flurry of heated reaction, a kind of moratorium 
was called, allowing time for interested parties to check on the 
current validity of the findings, as well as to consult about 
what action should be recommended. The Knesset called upon its 
Educational Committee to consider the Report, and the Ministry 
of Education formed an ad hoc committee to study the findings in 
m ore detail. For about two years, while these committees were 

•i a o wa q heard about the Van 
carrying out their assignments, little was 

Leer Study. 

The Knesset Educational Committee devoted four sessions to 
the Report, hearing testimony from various experts, including the 
research team. Then it appointed a subcommittee of six Knesset 
members, who worked for 27 months, holding eight formal sessions 
and visiting schools for disadvantaged learners. The subcommittee 
also issued invitations for external comments (i.e., from faculty 
members at schools of education or teacher training institutes, 
from the general public), but response was poor. Only one school 
of education team turned in a written comment, and about 10 Other 
people responded (representatives of private institutions, authors, 
and teachers), all of whom had their own axes to grind. On 
December 19, 1979, at a plenary session of the Knesset, the sub- 

~r,r-oH 19 "suaaestions for resolution" clustered 
committee presented 12 suggestion 

,»> -nrririilum (3) the learner 
into five groups: (1) resources, (2) curriculum, 

and his family, (4) teachers, and (5) principals. 
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The Ad Hoc Committee of the Ministry of Education consisted 
* of four meters: the Chief Scientist, plus three other Ministry 
officials responsible for monitoring the primary school system 
and implementing recommendations. This conmdttee in turn created 
six subject-area subcommittees (reading, mathematics, geography, 
science, biblical studies in nonreligious schools, biblical 
studies in religious schools) and a resource allocation subcom- 
mittee. The former (each consisting of five or six members, 
usually teachers and curriculum experts) examined test items to 
check their validity and adequacy. All but the science subcom- 
mittee were satisfied with the content validity of the items. 
Several of the subject-area subcommittees opposed the Report's 
suggestion that the curriculum was overloaded. All of them en- 
dorsed 'the establishment of a set of core requirements for each 
subject area, probably by way of ratifying a decision that had 
already been trade by the Standing Committee for Primary Education. 
The resource allocation subcommittee, after studying the data, 
recommended that, in the future, each ministry department be re- 
quired to report its activities in such a way as to indicate their 
contribution toward improving the situation in schools for the 
disadvantaged . 

Another important impact of the Van Leer Study was that it 
prompted additional research of two types: replication studies and 
supplementary analyses. The replication studies (carried out to 
see if the system had changed since 1973, when the Van Leer data 
were collected) covered two areas: resource allocation and 
achievement. In the first area, the Raziel Bulletin (1978) 
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indicated »at there had been increases in the resources allocated 
to both advantaged and disadvantaged schools, that the. latter had 
achieved parity on some resource variables, but that gaps still 
existed. This last finding prompted the Ministry of Education to 
initiate yet another study (Davis and Sprinzak, in process) which, 
though not yet completed, seems to indicate that further progress 
has been made in equalizing resources. In the area of achieve- 
ment , the Jerusalem District Achievement Survey (limited to the 
reading and mathematics achievement of fourth-graders in the 
Jerusalem School District) showed no improvement in mathematics 
achievement and only slight improvement in reading achievement 
since 1973. The second type of research (carried out between 
1977 and 1980) mined the data base in the Van Leer Study to 
throw light on issues not covered in the Report. For instance, 
cahan (1977) used the data to check for possible teacher discrimina- 
tion agarnst pupils of Asian-African origin, he found no evidence 
of discrimination. Peled's secondary analysis of intelligence 
test data (1980) found that the two e«hnic groups differed in 
their intellectual ability patterns. 

When the committees had completed their work in 1979, several 
further administrative and legislative steps were taken. The Ad 
H oc committee of the Ministry of Education pulled together the 
suggestions of its subcommittees and issued a document containing 
36 recommendations, which served as a basis for discussion at 
various forums within the Ministry. These recommendations were 
general policy statements, exhortative in tone, that did not 
really offer operational details, although they were directed at 



the specific departments which would be responsible for implement- 
ing them. The Educational Committee of the Knesset approved the 
recommendations of its subcommittee and presented them to the 
full parliament on January 1, 1980, asking that the Minister of 
Education respond within six months. On July 28, 1980, the 
Minister presented his response, outlining the steps that had been 
taken to implement the recommendations: allocating more resources 
to disadvantaged schools, establishing more adult education pro- 
grams, paying higher salaries to teachers of the disadvantaged, 
and setting core requirements. 

in a sense, the Van Leer Study has not been completed, since 
its effects are still being felt. People discussing particular 
educational issues still refer to the Report, and its influence 
on certain decisions made within the school system is clear. 
Researchers continue to mine the data base, and school adminis- 
trators continue to protest that its findings are not up-to-date 
and that further progress has been made. 

Achieving Impact 

Th e literature distinguishes several different types of 
evaluation use, including instrumental use (when evaluative in- 
formation is used directly in making programmatic changes) , con- 
ceptual use (when evaluative information influences policymakers' 
thinking without having any direct effect on action) , and symbolic 
use (when an evaluation is conducted for symbolic purposes: e.g., 
to satisfy the requirements of external agencies). In addition, 
one may distinguish various levels of use (e.g., the national 
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level, the school district level, the classroom level). Finally, 
the impact of an evaluation may be immediate and- limited or gradual 
and cumulative. The uses made of the Van Leer Study findings were 
primarily conceptual (and, to a more limited degree, instrumental); 
they occurred at .the highest level of the system; and the overall 
impact was gradual and cumulative. 

Several factors (some of them mentioned in the literature on 
evaluation utilization) seemed to enhance the impact of the Van 

Leer Study: ~~* / P 

l^jrecJnica]^ The Van Leer Report 

was generally praised for its technical quality and methodological 
rigor, even by those who criticized specific aspects of it. The 
research literature indicates that methodological rigor is a 
necessary but not sufficient condition for utilization. 

2 involvement of Prominent Scholars. The study was also 

accorded credibility because a large group of scholars-experts 
from both the Israeli and the international research communities- 
participated as ad hoc consultants at various stages, not only 
contributing directly to the quality of the study but also -Help- 
ing to assure its acceptance and to minimize sharp criticism 
against it. While the research literature frequently mentions 
report credibility as an important factor, most observers tend to 
see it as an extension of the personal credibility of the evaluator 

Characteristics of the Evaluators . One of the reasons 
the Van Leer Study received such attention was tAt the Principal 
investigator had acquired a national and international, reputa- 
tion as a dedicated researcher, a charismatic speaker, and an 



J 



3.3 



ardent advocate for hi, pedagogical ideas. The research litera- 
ture mentions the importance of the evaluated characteristics 
in influencing impact, usually focusing on such characteristics 
as credibility, orientation toward providing user information, and 
raP port with audiences. In this case, the personal dynamism of 
the Principal Investigator was the most outstanding evaluator 

characteristic. 

4^_jcnteres^ Since the Van 

k„ t-hP Ministry of Education, there 
Leer Study had been requested by the Ministry 

was every assurance that attention would be paid to its findings 
at the highest level. Moreover, the study addressed critical 
issues, topics of concern not only to policymakers and educators 
but also to the general public, this further assured that it would 
be highly visible. The evaluation literature notes both these 

factors as important. 

l ^ sss£ls; ^r 1 ^. The findings of the Van Leer Study 

„e r e consistent with the findings of another large-scale study 
(carried out by the prestigious Etzioni Committee, and thus gained 
additional credibility. As various observers have pointed out, 
an evaluation tends to be seen as valuable when it adds new 
dimensions to, or substantiates, data already available. 

t ^>^>.ts of Report Publication. Various events pre-^ 
ceding the release of the Report-including frequent progress re- 
ports, "leakage" of some of the findings at seminars, and the 

4- n f hundreds of teachers and thousands of 
direct involvement of hundreds oj. 

parents-contributed to the interest it aroused. 
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L __lnstituti^^ The existence 

within the Israeli Ministry of Education of the Office of the 
Chief Scientist-who is responsible for translating research 
findings into recommendations for action-further guaranteed 
that the findings would be utilized. Some observers have noted 
other such organizational mechanisms that facilitate impact. 

Several other issues arise in connection with the impact of 
the Van Leer Report and deserve some mention. 

First, there is some truth to the charge that the Report 
was oriented more toward the research community than toward the 
lay reader; but so are most reports of large-scale surveys. 
Perhaps such a study should produce several reports, aimed at 
different audiences and written in different forms and styles. 

Second, the overwhelming amount of information presented 
in the Report makes for difficult reading and created some con- 
fusion. Perhaps it would have been preferable to focus the study 
on a more narrow range of issues. Yet the very scope of the 
study, and the plethora of data it produced, carry certain ad- 
vantages: (1) some unexpected findings emerged about unknown 
phenomena; (2) the Report serves a "handbook" function, enabling 
the reader to extract information about emerging issues and to 
obtain baseline data on the details of broad topics; and (3) the 
peripheral data summaries help support the major findings. 

Third, the prodigious size of the Van Leer Report had several 
consequences not anticipated by the authors: (1) it laid the 
basis for a moratorium on action, delaying official acceptance 
(or rejection) of the recommendations but at the same time 
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allowing policymakers and educators more time to consider the 
findings and to think seriously about Israel's educational 
problems; (2) it allowed the data to be exploited to support con- 
flicting views; and (3) it allowed the data to be interpreted, 
in some cases, in ways. that directly contradicted the authors' 
intentions. The lesson to be learned here is that, when it comes 
to controversial issues, authors might be well advised not only 
to state what conclusions can be drawn from the findings but also 
to make explicit what conclusions cannot be drawn. 
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